Add Gymnasium's 11 MuJoCo environments, played from pixels, with a Modal baseline job - #31

What

All eleven of Gymnasium's MuJoCo -v5 environments, played from the rendered screen with one choice per step. This works like laya/controlgames.py does for classic control.

• laya/mujocogames.py: MujocoGame subclasses ControlGame, so it gets the same previous-frame ghosting, seeding and scoring. MuJoCo actions are continuous, so each game names a set of pushes:
  • InvertedPendulum / InvertedDoublePendulum push the cart (3 and 5 pushes), seen from a fixed side camera where a push right moves the cart right on screen.
  • Reacher, Pusher, Swimmer, Hopper, Walker2d, HalfCheetah, Ant, Humanoid, HumanoidStandup torque one joint at a time: NONE, then <JOINT>_POS / <JOINT>_NEG for each actuated joint. That is 5 options for Reacher up to 35 for Humanoid. Reacher and Pusher get closer cameras; the rest keep Gymnasium's tracking camera.
• Experts, using the same named pushes as the model:
  • Pendulums: a linear controller, and LQR from a finite-difference linearization rounded to the nearest push. Both hold the pole for all 1000 steps.
  • Seven torque games: a lookahead planner. It holds each push for a few steps in the simulator, restores the state, and picks the best return. A test checks that it leaves the episode unchanged.
  • Pusher and Humanoid: no planner I tried beat doing nothing, so they have no expert and no normalized score.
• Baselines: random_policy, and a new still_policy (always NONE), on the same seeds.
• laya/controlgames.py: small hooks (SPECS, _make, _env_action, play_episodes(make=)), and solved_rate is None for games without a threshold. Classic-control behaviour is unchanged.
• laya/games.py: goals and actions for the new games, generated from TORQUE_JOINTS. Existing questions don't change, so PROMPT_FORMAT_VERSION stays at "1".
• Recording and baseline: laya.mujocogames.record writes a real-time .webm with the model's probabilities beside the screen, and baseline scores a checkpoint on one game. examples/mujoco_video.py and examples/mujoco_baseline.py run them locally.
• modal_app.py::mujoco_eval: runs every game at once, one L4/A10G each (play_mujoco, rendering with OSMesa), and writes the videos and baseline.json locally. _load_policy_agent now accepts a Hub revision.
• Docs: a "Play the MuJoCo environments" section in site-docs/how-to/play-games.md.

This PR doesn't add the MuJoCo games to the games suite and publishes no numbers.

Known issue: Humanoid prompts don't fit the question budget

At the released checkpoint's head_max_len=256, the 35 options of Humanoid and HumanoidStandup are cut to 5 tokens each, which leaves only 7 distinct option texts: ABDOMEN_Z_POS: torque the abdomen z one way (positive) becomes - ABDOMEN. Shorter wording can't fix it, because even RIGHT_SHOULDER_1_POS alone is longer than 5 tokens. The fix is a smaller action set for these two games (for example grouped limb moves) or a larger budget. That decision is still open, so the Humanoid rows below are not valid.

The other nine games keep every option distinct. Pusher (15 options), Walker2d and HalfCheetah (13 each) and Ant (17) lose part of each description, but no action names.

Zero-shot baseline (not a published number)

modal run modal_app.py::mujoco_eval --revision 8b318c99d7ad3ce19c24369263463882eada9d1e --max-steps 0: thaitea/laya-vision@8b318c9, 3 model episodes and 10 of each baseline from seed 200000, full-length episodes, about 26 minutes on Modal.

⚠ These rows are invalid: see the known issue above.

On the nine valid games, the model beats random on 7 and never beats doing nothing. Normalizing against random flatters it: on the robots that only need to stay upright, NONE is the stronger baseline. Ant's 0.75 is mostly survival, since the planner barely beats doing nothing there.

Testing

• tests/test_mujocogames.py (skipped unless mujoco and gymnasium are installed): for every game, pushes match the action names and action space, and a frame renders. Experts beat random and still. The lookahead leaves the episode unchanged. Pendulum experts hold the pole to the limit. Also covers seeding, ghosting, and the questions. 34 passed together with tests/test_controlgames.py.
• The full python -m pytest -q run on the first commit gave 304 passed and 6 tests/test_smolvlm2.py ImportErrors, which also fail on main in this sandbox.
• mujoco_eval ran end to end on Modal: a 2-game probe, then all 11 games.
• mkdocs build --strict passes (pip MkDocs; nix isn't available in the sandbox).

🤖 Generated with Claude Code

https://claude.ai/code/session_01X7wWfNMM9Rbk1757w3FaTH